Discrete Element Method (DEM) has been crucial in understanding complex dynamics of granular media, for example sand, rocks and powders. Simulations typically require a setup phase, where the initial sample for the simulation is generated. In case of railway tracks, for example, ballast being the granular media, is generated and compacted after free fall onto the ground, and train passage on top of it is simulated right after. The setup phase, a significant part in this simulation, usually takes 4-5 hours for 5000 grains (compacted layer with dimensions: 1.4 m length, 1.2 m width and 0.3 m height). Hence, this approach becomes prohibitive when DEM is used to generate large representative samples. This paper proposes a method to rapidly generate large samples of granular media- for example, cross sections of ballast layers, bead samples, and compacted sand samples. The method relies on diffusion models that learn to generate the realistic samples, based on pre-existing databases of samples that were generated with smaller spatial dimensions. From these, representative patchified areas for the geometrical properties like compactness, size distribution among others, are extracted and used as inputs to standard Denoising Diffusion Models (DDMs). The method first generates new patches in batches and arranges them on a long checkboard grid (Figure 1a). These patches are then used for conditional generation of adjacent areas, imposing spatial compatibility. Notably, StableDiffusion can generate coherent adjacent patches without any additional fine-tuning for the conditional generation (Figure 1b). Segmentation of the generated images is performed using watershed algorithm and later converted into vector inputs to DEM software such as LMGC90 (Figure 1c), where it can be simulated as wanted (freefall example in Figure 1d). To verify the realism of the generated samples, statistical analyses confirm that both particle shapes and their spatial distributions are accurately represented. This approach enables the generation of large samples of granular media in a fraction of the time it usually takes for the DEM software to produce the samples of same size (Figure 2). Larger samples require linearly more computational time, compared to exponential time increase often observed with traditional DEM.